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ABSTRACT 



A curriculum evaluation approach has been proposed that can 
enhance educators' understanding of how curriculum designed to effect social 
and vocational change in market economy cultures interacts with students, 
schools, and society. This method combines elements of a social evaluation 
approach fashioned by M.W. Apple and L.E. Beyer (1985) with a critical 
ethnography model developed by P.F. Carspecken (1995) . It includes five 
levels or basic categories of inquiry. Stage 1 analyzes the social , 
historical, and ideological context into which the program is introduced and 
how that context is currently influencing school reform. It reveals the 
ideological connections between dominant social forces and school reform and 
identifies and critiques the various assumptions underpinning vocational and 
social preparedness programs. Stage 2 identifies informal components of 
employability skills curriculum to consider their potential impact on 
students and society. Stage 3 requires a conceptual analysis of the 
employability skills discourse contained in the formal curriculum of 
programs, such as the Career and Personal Planning curriculum. Stage 4 uses 
an ethnographic collection of data to acquire knowledge of the dynamic 
interaction between curricula and classroom culture. Stage 5 analyzes 
findings from the previous four stages through a critical conceptual 
framework founded on principles of social justice and equality. (Contains 23 
references.) (YLB) 
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Public schooling in virtually all industrialized countries faces growing pressure from the 
corporate sector to adopt occupationally relevant cmriculum (Crouch, Finegold & Sako, 1997). On a 
provincial level, the Career and Personal Planning (CAPP) curriculum, introduced into B.C. 
secondary schools in 1995, reflects this current trend toward vocational education reform. The career 
preparedness component of CAPP allegedly strengthens labour market opportunities for B.C. 
secondary school students by equipping them with a set of generic employabiUty skills. In spite of the 
present popularity of employability skills curricula, however, little formal research has been 
conducted to assess the moral appropriateness, social impact, or conceptual soundness of 
transforming Canadian schools in this fashion. This paper takes a preliminary step to correct this 
deficit by proposing a method to investigate the moral, social and educational consequences of 
curricula implemented for social and vocational purposes. 

Curriculum evaluations of vocational education programs typically adopt a systems 
management approach that includes four basic stages of development: 1) selecting evaluation targets, 
i.e., determining what is to be measured and how; 2) planning the evaluation, i.e., determining data 
collection methods; 3) collecting and analyzing the data; and 4) preparing the evaluation report 
(Chandler et. al., 1997; Vannatta et. al., 1998). Systems management evaluations assume the 
effectiveness of a system, i.e., employability skills curricula, can be evaluated on the basis of how 
closely the system output matches system objectives (Apple, 1990). Although these evaluations 
promote the achievement of curriculum objectives by highhghting a program’s instrumental 
inadequacies, their functionalist format severely restricts the scope of critical inquiry. Systems 
management evaluations determine whether curriculum objectives have been achieved by the 
program’s recommended methods, for example, but do not assess whether those objectives, 
especially when placed within the larger social context, can be morally, conceptually, or 
educationally justified. 

The inductive research methods employed in systems management evaluations produce 




3 



Alternative Curriculum Evaluation 



3 



impressive statistical charts and data, but explain little, if anything, of the social structures from 
which that data emerges. The positivist tradition supporting this approach to curriculum evaluation 
requires education researchers to use methods and terminology developed by the natural sciences. 
Within inductive education research, for example, there is an overriding emphasis on the noble and 
impressive scientific research requirements of reliability, validity and generalizabihty. Since scientific 
discourse is generally afforded a higher epistemic status in education than the discussion of ethical or 
moral matters, system management evaluations are exceptionally popular, while foundational 
questions associated with curriculum reform, regardless of their tremendous moral importance, often 
remain ignored (Apple, 1990). By neglecting the social structures and ideological struggles that 
underpin curricuhun development, then, these evaluations fail to investigate the impact of social 
engineering programs on students outside stated objectives. 

In this paper, 1 conceptualize curriculum evaluation in a different fashion from how it is 
conceived within system management methodology. 1 hope to provide educators with an evaluation 
approach that can enhance our understanding of how curriculum designed to effect social and 
vocational change in market economy cultures more fliUy interacts with students, schools, and 
society. This proposed method combines elements of a social evaluation approach fashioned by 
Apple and Beyer (1985) with a critical ethnography model developed by Carspecken (1995), and 
includes five levels or basic categories of inquiry: 1) an analysis of the social, historical and 
ideological context into which the program is introduced, and how that context is currently 
influencing school reform; 2) exposing the informal curriculum accompanying the educational 
change, and reviewing its potential impact on students and society; 3) a conceptual analysis of the 
formal curriculum; 4) an ethnographic collection of data to acquire knowledge of the dynamic 
interaction between curricula and classroom culture; and 5) an analysis of the findings from the 
previous four stages through a critical conceptual framework founded on principles of social justice 
and equahty. The five stages of analysis are not apphed in rigid chronological fashion, however, but 



er|c 



4 



Alternative Curriculum Evaluation 



4 



rather in an overlapping, on-going and interconnected format. 

The analysis of context in stage one is designed to reveal the ideological connections between 
dominant social forces, i.e., corporations, and school reform, and to identify and critique the various 
assumptions underpinning vocational and social preparedness programs. These ideological 
assumptions reflect moral, ontological and social viewpoints subject to evaluation through the critical 
lense provided in stage five. Evaluations limiting their inquiry to schools and programs, while 
ignoring social context, remain relatively unreflective about interests, values, and ideologies in 
curriculum: 

If our unit of analysis is only the school, the issues surrounding 
curriculum evaluation can stand alone and less of a serious challenge 
can be made against the process/product path it has taken. If, 
however, the school is interpreted as inextricably connected to 
powerful institutions and classes outside itself, then our unit of 
analysis must include these connections (Apple and Beyer, 1983, 
p.427). 

Since schools are not immune to social pressures, disregarding social context and the connected 
ideological assumptions precipitating curriculum reform, means that evaluators accept the problems, 
values and objectives of education are those identified by dominant social interests. Further, the 
subsequent shift from the macro to the micro level of analysis should reveal how local education 
practice is influenced by global market forces, and how theory translates into classroom experience 
in social engineering programs (Taylor, 1998). 

A brief analysis of the present social context, and its direct impact on CAPP, reveals the 
influence economic forces currently exact on public education policy by imposing market economy 
ideologies on schools. The Conference Board of Canada (CBOC) is the central lobbying force for 
domestic private sector business interests in public education pohcy development. The organization 
funds two education councils, the national council and the corporate council, both entirely dedicated 
to influencing Canadian pubhc schooling. The corporate council is comprised of senior executives 
from CBOC member companies, including a disproportionate number of executives from large 
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technology related corporations (Taylor, 1998). The more powerfijl corporate council also developed 
the influential Employability Skills Profile (ESP), a list of generic skills employers supposedly 
require in students they hire (CBOC, 1997). 

The CBOC’s impact on Canadian public education pohcy has been both widespread and 
profound. The board reports, for example, that business-education partnerships are in “explosion 
mode” with twenty thousand now in place across the country (as cited in Robertson, 1998, p.279). In 
Alberta, numerous programs already have been undertaken based on CBOC initiatives (Taylor, 

1998). In British Columbia, the CBOC plays an authoritative role in the career preparedness 
component of CAPP, as students are expected to master ESP skills before graduating from secondary 
school (CAPP, 1995).The CBOC’s stated interest in education is to engage business and schools “in 
partnerships that foster learning excellence to ensure that Canada is successful and competitive in the 
global economy” (CBOC, 1997). From its business-driven perspective, the concept of “success” is 
inextricably connected to attaining the quantifiable rewards consistent with the consumerism of 
market economy cultures. A CBOC pamplet. Matching Education to the Needs of Society ( 1 995), 
reflects the board’s ftinctionalist education agenda by asking, “Is the present [education] system 
capable of preparing students for the challenges of the 2 P* century and for a working hfe that is 
characterized by high technology and rapid change?” Kuehn (1997) has challenged the moral, 
ontological and educational assumptions entailed in corporate schooling objectives for regarding 
students as human capital, and marginalizing the social, cultural and ethical responsibihties of 
schools. Regardless of whether one agrees with Kuehn’ s analysis, highlighting the connections 
between dominant social interests and education exposes the market economy assumptions propeUing 
current vocational education reform to critical review. 

A common assumption on which employabihty skills programs are predicated is their 
hypothesized ability to improve labour market opportunities for students (CAPP, 1995; CBOC, 

1997). To evaluate whether this claim is warranted once again requires more than a mere review of 




6 



Alternative Curriculum Evaluation 



6 



student achievement scores or curriculum delivery protocols. Rather, in the case of CAPP and other 

similar programs, the contextual phase dictates an analysis of current labour market conditions to 

determine whether individual skill deficits are a major contributing cause to unacceptable 

unemployment levels. In fact, a prehminary investigation of labour market circumstances suggests 

that skill deficits have little to do with present imemployment levels. Livingston (1996) explains: 

Some empirical research indicates that an adequate Grade 8 education 
is all that is needed to perform the typical factory or office job in 
advanced industrial societies. Further studies reveal that since the 
early 1970s at least a third of the employed North American 
workforce have work-related skills they could use in their jobs but are 
not permitted to use (p. 76-77). 

From an ideological perspective, then, emphasizing educational shortcomings and individual skill 
deficits as the cause of unemployment problems successfully deflects attention away from their actual 
systemic origins. 

Stage two of this evaluation method identifies the informal components of employability 
skills curriculum to consider their potential impact on students and society. Schools are generally 
structured to dehver specified pre-packaged knowledge to students, and thus exercise considerable 
epistemic authority over them. Coupled with the epistemic legitimacy afforded to schools as 
necessary socializing institutions, education provides the perfect means to impart dominant values 
and attitudes in future citizens (Althusser, 1973; Bourdieu & Passeron, 1994). Conveying ideologies 
through curriculum, however, need not reflect a large scale conspiracy on behalf of corporate elites, 
but only the informal, largely unspectacular, transmission of prevailing, often widely accepted, 
assumptions, beliefs and values to students (Portelli,1993). According to Habermas (as cited in 
Eagleton, 1991), for example, ideology typically takes the form of commonly accepted 
communication patterns systemically mitigated by social power relations. By transmitting and 
legitimating dominant attitudes and values to students, the employability skills discourse provides an 
excellent example of how schools convey ideology to students. 
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In the career preparedness section of CAPP, the informal curriculum transmits various norms 
and values not stated in the formal curriculum document. By presenting the concept of employability 
skills to students from a functionahst perspective, for example, CAPP legitimates the existing 
economic paradigm to students. CAPP’s functionalist format encourages students to conform 
passively with market economy values and practices rather than actively assessing their general 
acceptahihty. Since their primary stated objective is enhancing student employment prospects, 
employability skills programs also contain the tacit premise that high levels of joblessness result from 
individual deficits rather than from structural inequalities of opportunity, an indispensable myth in 
promoting market economy culture. Once again, responsibihty for unemployment is deflected away 
from its systemic origins, and the market economy practices inflicting widespread job loss are 
insulated from moral critique. 

An analysis of the informal curriculum related to social engineering programs should also 
explore the particular forms of knowledge the formal curriculum validates. Apple and Beyer contend 
that “. . . prior to measuring whether or not students are able’ to learn or have learned a particular 
set of facts, skills, or dispositions, we should want to know whose knowledge it is, why it is 
organized and taught in this particular way, to this particular group” (p.43 1). A review of CAPP 
reveals a complete absence of labour movement history, or indeed any suggested discussion on the 
role and purpose of labour imions, while corporate attitudes, values and beliefs dominate the 
document. As part of the informal curriculum, then, - in this case what is left out of the formal 
curriculum - working class knowledge and experience are excluded, while corporate knowledge is 
legitimated and conveyed to students. 

Stage three of this evaluation approach requires a conceptual analysis of the employabflity 
skills discourse contained in the formal curriculum of programs such as CAPP. The purpose of this 
evaluation phase is twofold: First, many of the so-called skills in the work experience component of 
CAPP are not skills at all, but are more appropriately categorized as attitudes, values and 
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dispositions. Under ESP’s heading of personal management skills, for example, students are 
expected to demonstrate “a positive attitude toward change” (CBOC, 1997). Identifying such 
attitudes as skills confuses important conceptual distinctions between the two, and may preempt 
moral evaluation of the former’s educational appropriateness. Similar to the underlying assumptions 
supporting employability skills programs, the values, attitudes and dispositions, once properly 
classified, are assessed through the critical framework identified in stage five. Secondly, even if the 
various assumptions supporting employability skills education are judged morally acceptable, serious 
questions remain regarding the pedagogical efficacy of these programs. A conceptual analysis 
pointing out fundamental category mistakes that impede student achievement of desired cognitive 
competencies may reveal that employabihty skills programs are not only morally suspect, but 
instrumentaUy flawed as well. 

Analyzing context, identifying and critiquing messages conveyed to students by the informal 
curriculum, and conceptually analyzing the skills discourse in the formal curriculum are important 
steps in the assessment process, but an examination of the dynamic interplay between curriculum and 
classroom is also required. The first three stages provide insight into the potential interaction between 
students, schools and social structure, but structural analyses alone cannot reflect the complexity, 
imcertainty and impact of human agency on that interaction. Apple and Beyer explain: “No social 
institution, no set of ideological forms and practices, is ever totally monolithic. Students will not 
necessarily accept what die school teaches and we cannot take for granted that students or teachers 
are passive vessels who imcritically accept what curriculum documents entail” (p.432). 

Ethnographic research conducted by Wdhs (1973), Wertsch (1998), and Lave and Wenger 
(1996) highhghts the way in which human agency mediates between individual cognition, and the 
cultural, institutional and historical context of schools. Collectively, these findings seriously challenge 
exclusively structural critiques of education, and suggest the need for some form of etlmographic 
study to acknowledge the complex interchange between curriculum, student culture, and individual 
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cognition. From a reconstructionist perspective, structural critiques that fail to recognize human 
agency offer little hope of transforming the repressive and reproductive schooling practices they 
identify. Indeed, evaluations ascribing agency to corporations and economic systems rather than to 
teachers and students, and remaining aloofly detached from classroom settings, are themselves 
socially reproductive. No critical curriculum evaluation is complete, then, without investigating what 
is taught in the classroom, and considering student reaction to that instruction. 

Employability skills programs provide ample opportunity for both teachers and students to 
apply the curriculum in various unexpected ways. Although critical thinking has been appropriated 
into the employability skills discourse (Lankshear, 1997), for example, no procedural parameters are 
typically established for its classroom application. Ethnographic research indicates that even when 
the supply of learning tools is regulated, their actual classroom application cannot be completely 
controlled (Werstch, 1998; Lave & Wenger, 1996). Ironically, by adopting critical thinking as an 
objective, programs such as CAPP provide the necessary intellectual tool to challenge their own 
credibility as teachers and students could employ the concept to challenge the assiunptions 
supporting employability skills instruction. The more salient point, however, is that only ethnographic 
research can identify and report curriculum consequences not knowable through structural analysis 
alone. 

The ethnographic stage begins with compiling a primary record, or thick description, of 
classroom activities where virtually all speech acts, body movements and postures are recorded to 
represent as fully and accurately as possible the interaction between the curriculum and the 
classroom. Conceived by Eisner (1977), the concept of thick description emphasizes the importance 
of context in understanding student behaviour by “describing the meaning or significance of 
behaviour as it occurs in a cultural network saturated with meaning” (p.97). The second step of stage 
four requires speculating on the meanings of events and actions recorded during the compiling of the 
primary record, a process Carspecken (1996) refers to as “preliminary reconstructive analysis” 
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(p.93). During the reconstructive phase, because of its interpretive quahty, a certain degree of 
subjective analysis occurs, but it attempts to identify behaviour patterns that offer insight into both 
classroom and student response to the program. Carspecken also advocates a stage of dialogical data 
generation, where students and teachers contribute directly to the data collected through various 
quahtative research techniques. Their participation is designed to check possible researcher bias, add 
to the interpretative data, and ensure that teachers and students are not objectified by the 
etlmographic evaluation process. In an assessment of CAPP, case studies, interviews or focus groups 
could be employed to gauge more fully teacher and student reaction to the program. 

Social research suggests that society continues to remain firmly structured along race, class 
and gender lines (BaUantine, 1997), and the actual chasm between social classes has grown 
considerably over the last decade (Robertson, 1998). As a matter of moral coherence, social 
stratification, and any schooling practices operating to reproduce it, ought to be considered ethically 
inappropriate in a society embracing egalitarian principles. Indeed, the school is rightly conceived of 
in our society as an institution expected to advance the democratic concept of social justice (Apple, 
1990). From a social justice perspective, then, the ultimate goal in any critical curriculum evaluation 
is determining the degree to which the program under review amehorates or worsens social 
stratification and injustice. 

In an attempt to address this question, stage five introduces concepts like social justice, class 
structure and gender equality to provide a moral conceptual framework to evaluate the data gathered 
during the other four stages. Following Apple and Beyer, and Carspecken, this stage of critical 
analysis contains the fundamental assumption that the principle role of education research is 
challenging all forms of social oppression reproduced through schooling practices. This final stage 
attempts to intermesh the results from the previous four stages into an existing macro critical theory 
of social explanation to evaluate the moral appropriateness of programs introduced for social and 
vocational reasons. Consistent with critical approaches, the level of inference in stage five increases 
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dramatically as the findings fi-om the first four stages are explained with reference to a social system 
theory. As Carspecken suggests, “a critical researcher is able to suggest reasons for the experience 
and cultural forms reconstructed having to do with class, race, gender, and political structures of 
society” (p.43). Within a neo-Marxist critical fi-amework, for example, the evaluation findings could 
he interpreted on the basis of whether they serve the needs of the social elite or actually improve 
social and vocational opportunities for disadvantaged students (Ballantine, 1997). 

In this paper 1 have provided an alternative critical evaluation approach designed to highlight 
the individual and social consequences of education programs introduced to effect social and 
vocational change. 1 have argued that traditional systems management evaluations of such curricula 
are woefully inadequate because th^^ ignore fundamental moral, social and ontological assumptions 
supporting such programs. In some instances these assumptions may reproduce forms of systemic 
injustice that are inconsistent with the general moral objectives of an egalitarian society. In response 
to this methodological shortcoming, I have proposed a five stage critical evaluation approach that 
examines social context, investigates informal curriculum content and consequences, and 
conceptually analyzes the employabdity skills discourse within the formal curriculum. Seeking to 
avoid an entirely structural critique of the co nn ections between society, schools and students, stage 
four proposes an ethnographic study to reflect the dynamic interaction between the curriculum and 
the classroom. Finally, the data collected during the first four evaluation stages is situated within the 
context of a macro critical theory, and evaluated on its moral consistency with fundamental principles 
of social justice. 
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